Evaluation of students’ digital animated multimodal narratives and the identification of high-performing classrooms

Contemporary approaches to literacy embrace digital and multimodal communication, and this is increasingly recognised in the syllabi prescribed by various education authorities across the world. Insufficient attention has been given to the evaluation of multimodal texts in ways which are semiotically grounded, accessible to the teacher and scalable to larger research studies. We present an evaluation instrument that addresses these requirements. The application of this instrument to 81 texts drawn from 17 classes has established the viability of the approach and allowed a subset of ‘high achieving’ classes to be identified. The derivation of the instrument is described in detail, the final form presented, evaluator guidelines elaborated, and the rating scales developed in full. Limitations are discussed along with recommendations for further work and development, but as an evaluation initiative the current work is presented as an important contribution to the continued development of multimodal pedagogy. Journal of Literacy and Technology 82 Volume 13, Number 3: December 2012 ISSN: 1535-0975 The reconceptualization of literacy in the context of our increasingly digital, multimodal information and communication world is now becoming more widely and prominently recognized in the curriculum requirements of government education authorities (England 1999; Australia 2009; Singapore 2010). With electronic, multimodal texts not only the dominant and preferred medium of today’s digital generation, but also a required aspect of mandated curriculum documents, teachers in various ways have been weaving multimodal literacy into students’ interpretive textual experience and to a somewhat lesser extent, into their text creation experience. However, while new digital multimodal literacies pedagogies are evolving (Anstey and Bull 2006; Unsworth 2008; Mills 2010; 2011), relatively little attention seems to be given to the development of an appropriate approach to the assessment of multimodal literacy development (Unsworth and Chan 2008; Unsworth and Chan 2009), especially in relation to students’ creation of multimodal texts (Baxter and Simpson 2008; Kimber and Wyatt-Smith 2008; Macken-Horarik 2008). A very general conceptualization of an approach to evaluating dynamic digital and filmic texts produced by junior high school students in response to literature in the English classroom has been devised by Kimber and Wyatt-Smith (2008). They see the evaluation of such texts at the intersection of the textual evidence for students’ e-proficiency (skill in the utilizing software affordances) and the quality of the textual content, cohesion and design. While this is a useful framework there is no specification of criterial textual features that would differentiate quality in respect of the four dimensions, and from the descriptive accounts of them it is difficult to clearly distinguish between cohesion and design features. On the other hand, working with primary school students’ production of claymation, stop-motion movies, Mills (2011), following the work of Andrew Burn (Burn and Parker 2003; Burn and Leach 2004; Burn and Durran 2006), has related the meaning-making affordances of filmic text production to the “grammar of Journal of Literacy and Technology 83 Volume 13, Number 3: December 2012 ISSN: 1535-0975 visual design” extrapolated by Kress and van Leeuwen (2001/2006) from Hallidayan linguistics (Halliday 1978; Halliday and Matthiessen 2004). This enables a specific textfocussed, differentiated analysis of the students’ movies providing the basis for feedback on learning and guidance to teaching. A somewhat similar approach was used by Thomas (2008) to discuss the quality of machinima produced by primary school students. However, this work has not extended to the formulation of a validated consistent procedure or specific instrument for systematically evaluating students’ filmic texts. This paper addresses the evaluation of digital animations in the context of our work facilitating middle years students’ creation of 3D animated narratives (Chandler, O’Brien and Unsworth, 2010). Authoring narratives using 3D animation involves the students using computer software to create a movie in many respects similar to movies created using liveaction work with a video camera. To write/create using 3D software involves harnessing systems of choices for making meaning. The complete repertoire of meaning-making resources available in 3D multimedia is quite simply vast, framed (for example) by Cope and Kalantzis (2009) in terms of five modes: linguistic, visual, spatial, gestural and audio. To support students’ ongoing development of 3D animation authoring and to determine the most efficacious teaching practices, we need to derive a means of assessing the effectiveness of students’ deployment of these multiple meaning-making resources. We have sought an instrument that teachers and researchers can apply systematically and relatively quickly in responding to students’ work, providing informative feedback, and which could be scaled to provide systematic evaluation of several hundred texts. In this paper, we outline our approach which (a) attends carefully to intra-modal meaning (b) has the capacity to attend to intermodal meaning (c) is suitable for the evaluation of relatively briefs texts developed by young, inexperienced authors and (d) can be readily applied to the bulk analysis of texts. Journal of Literacy and Technology 84 Volume 13, Number 3: December 2012 ISSN: 1535-0975 The centrepiece of this article is the development of the evaluation instrument itself, the use of which is demonstrated through the identification of ‘high performing’ classrooms. The identification of such classrooms is important so that subsequent work can draw on other observational and case study data in order to explicate features of teaching and learning which are important in the creation of high quality multimodal texts by school age students. The discussion proceeds as follows. Firstly, the framework for an evaluation instrument is described in principle, followed by a presentation of the particular evaluation instrument used in our investigation. The detail of how that instrument was used to broadly discriminate between the quality of work from 17 classes is then presented. The paper concludes with a discussion of the efficacy of the approach and implications for future development and application. Evaluation by attending to semiotic systems The starting point for the evaluative approach we are advancing is to consider the systems of choices that a creator of multimodal texts makes. A simple description for still images of the manner in which various design elements (or codes) and conventions together form a system from which combinatorial selections are made to convey meaning was provided by Anstey and Bull (2006, p. 108) and is shown in Table 1. Table 1: Design elements and conventions in still images combine to make meaning (Anstey & Bull, 2006, p. 108) The design elements of are combined through the conventions of to make meaning • Colour • Texture • Shape • Form • Balance among design elements • Layout (how attention is attracted and focused) • Vectorality (how the eye is led through the image) We adapted this description to form a ‘template statement’ as the basis of our approach to Journal of Literacy and Technology 85 Volume 13, Number 3: December 2012 ISSN: 1535-0975 evaluation the text makes meaning by attending to [category of meaning] by strategic deployment of conventions associated with [design element] For instance, “the text makes meaning by attending to still images by strategic deployment of conventions associated with colour”, and separately, “the text makes meaning by attending to still images by strategic deployment of conventions associated with texture”, and so on. Thus, an evaluation of a whole text can proceed by attending to relevant semiotic systems, and then to the codes within each system. Using a range of such evaluative statements scopes the task of evaluation by identifying the semiotic systems that should be considered, and degree of delicacy to which they should be addressed. The form of the evaluation (e.g. written comment, yes/no, rating scale) is a separate consideration and our approach is addressed later in the section Towards an Evaluation Instrument. It should be observed that the template statement does not specifically include the conventions: it is assumed that the evaluator is sufficiently familiar with the relevant semiotics, although evaluator guidelines have been developed to facilitate confirmation of this for our purposes (see Appendix 1). An evaluator needs to work within the context of the social purpose of the text (in our case, a narrative piece, dealing with unusual or problematic events and their outcomes) and the socially constructed nature of conventions involved. For instance, black is the colour of death in some cultures, where as white carries that value in others; red conveys particular meanings in some cultures, but less so in others. Similarly, size, shape, proportions, clothing, hairstyle of characters will communicate important information. There are choices of colour, props, clothing and actual location which will situate the text in a particular time of day, season or era. Special effects such as fog can be used, and may variously signify a spooky environment or evening closing Journal of Literacy and Technology 86 Volume 13, Number 3: December 2012 ISSN: 1535-0975 in. In short, the approach we advance assumes that the evaluator is semiotically knowledgeable. A list of evaluative statements is not intended to be either exclusive or definitive. The intention is to scope the evaluative effort – items ‘to attend to’ – not to provide a checklist that of items that must always be covered and never exceeded. For instance, by listing ‘colour’ as one of the evaluative statements does not imply that black-and-white line drawings would be automatically criticised – the evaluator can attend to the matter of colour and make a judgement that it is not relevant in this case. Rather, in thinking through how adequately the text addresses the meaning communicated through still images, the approach ensures that considerations of colour are not overlooked. Thus, the intention is that evaluative statements would help structure an overall evaluation of a text as an entire and coherent communicative enterprise. Later in the discussion, we will also note some examples of how authors/creators can attend to some of the design elements in particularly creative ways which belie treating the design elements or conventions as a mere checklist. A further observation of this approach is that it tends to treat each semiotic system independently. If, for instance, a voice-over or background music were provided to accompany a still image then the design elements of each system would be treated separately, and there would be a risk of inter-semiotic meaning not being addressed. We need to identify this limitation from the outset and we have included a separate means for recognising those (see section on Other Considerations). Having discussed, in general terms, an approach by which an attention to the multiple systems of meaning and the design elements thereof can be used to frame an evaluation, we now turn to the application of this approach to those systems of meaning available to the 3D multimodal author and readily used by young, inexperienced authors/creators. Journal of Literacy and Technology 87 Volume 13, Number 3: December 2012 ISSN: 1535-0975 Semiotic systems and codes identified from the software environment Whilst this paper contributes to the general endeavour of improving the evaluation of multimodal digital texts, such texts can only be realized through the affordances of the particular (software) tools used to create them. The evaluation needs to take this into account, and is shaped, to some extent, by the functionality and capabilities of the software tool. Necessarily, we draw on an intimate knowledge of a particular item of software, Kahootz (Maggs, 2008), but in a way that would readily transfer to other products currently in the marketplace. We also attend to the systems of choices that are relevant to the type of text regardless of the system used to produce it, which could also include techniques such as live action filming, stop-motion animation or claymation. Through close attention to the software functions available, two broad systems of choices can be identified which map onto two of the systems of meaning-making within Hallidayan linguistics (Halliday & Matthiessen, 2004): the ideational and interpersonal. The following presentation of the development of the template evaluation statement for the respective systems and codes has been informed by an insider’s perspective of the software and references to key elaborations of the Hallidayan framework in relation to still image, moving image and sound (Kress & van Leeuwen, 2001/2006; van Leeuwen, 1996; van Leeuwen, 1999). The ensuing discussion concludes with a consideration of the third of the Hallidayan systems, textual meaning (or compositional meaning), and presentation of ‘other considerations’ which have been included in our instrument. That leads into a subsequent discussion of how these various components have been fashioned into a workable instrument by considering the issues of text-based evaluation, unit of analysis and a rating scale. 2 Further examples of software of this type include Muvizu (http://www.muvizu.com/), Kids Movie Creator (http://www.kids3dmovie.com/en_01/Products.aspx), Alice and Storytelling Alice (http://www.alice.org), Moviestorm (http://www.moviestorm.co.uk/hub/australia), Reallusion (http://www.reallusion.com/) and Anim8or (http://www.anim8or.com). Journal of Literacy and Technology 88 Volume 13, Number 3: December 2012 ISSN: 1535-0975 ‘Creating a world’: ideational and textual meanings The representational/ideational system is concerned with communicating the nature of events, the objects and participants involved and the circumstances in which they occur (Unsworth, 2001, p. 18). As we proceed to elaborate, ‘creating a world’ principally involves these systems of meanings. Using Kahootz, a 3D multimodal text is developed and presented as a series of scenes. The author selects one of many worlds (or sets) on which this scene is then further developed. It is not possible to import additional worlds, so one is constrained to work with a base palette from the library of worlds, but with the capability to re-colour or re-texture (i.e swatch) modify or appropriate them for a range of purposes. For instance, what initially appears as lush grasslands can be re-coloured to be a sparse desert. It is further possible to move through the world and thus choose a different location from the initially presented one. Each scene can be populated with a range of objects, which can be selected from the extensive in-built library (as it the case with ‘worlds’, it is not possible to import characters into Kahootz). The object can be re-sized, have its proportions changed, and aspects of each object can be re-swatched. Thus, following the initial choices about setting and location, the author must choose how to populate the world, a task that embraces set dressing, props and characters. Included within this are decisions about how the objects are physically positioned, as it would be possible to have these (appear to) float in mid air or be (partially) buried in the ground. Furthermore, there are choices related to the arrangement of these objects – showing a group of characters who are looking at each other to represent a conversation, for instance. Objects can be animated – that is, caused to perform built-in actions or move from one location to another. In addition to adding visual objects to a scene, audio can be added. Kahootz, for Journal of Literacy and Technology 89 Volume 13, Number 3: December 2012 ISSN: 1535-0975 instance, contained an extensive library of sound effects, along with the ability to record and import sound and together with the capacity for manipulation by specifying volume, pitch, echo, tremolo and duration. The audio mode, therefore, is a design element alongside choice of world, physical positioning, swatching, etc. From the preceding discussion, it should be clear that there are three ‘categories of meaning’ which contribute to the ideational system: • Setting and location (for instance, selection and swatching a set, identifying a location, adding various objects as set dressing, choosing lighting and special effects and including background sound effects or music) • Participant selection and construction (for instance, selection, swatching and sizing participants and including dialogue) • Arrangements and interaction of participants (for instance, the positioning of participants, the eyelines and gestures used) These are presented in Table 2, along with design elements to which one might reasonably attend. Table 2 is intended to be read in the form of the evaluative template statement, for instance: “the text makes meaning by attending to setting and location by strategic deployment of conventions associated with temporal location”. The text makes meaning by attending to ... by strategic deployment of conventions associated with ... Setting and location Choice of location Mood and atmosphere Temporal location (time of day, season, era, etc) Material location (environment, objects, inhabitants, etc) Material processes (animation of environment, objects, inhabitants) Narration &/or dialogue Sound effects Journal of Literacy and Technology 90 Volume 13, Number 3: December 2012 ISSN: 1535-0975